Papers with annotation task
Zero-shot prompt-based classification: topic labeling in times of foundation models in German Tweets (2025.acl-srw)
Copied to clipboard
| Challenge: | Recent advances in NLP have enabled the use of text-to-text annotation without providing training samples. |
| Approach: | They propose a text-to-text interface for automatic annotation using written guidelines without providing training samples. |
| Outcome: | The proposed approach is comparable with the fine-tuned BERT but without any training data. |
FITAnnotator: A Flexible and Intelligent Text Annotation System (2021.naacl-demos)
Copied to clipboard
| Challenge: | In this paper, we introduce FITAnnotator, a generic web-based tool for efficient text annotation. |
| Approach: | They propose a generic web-based tool for efficient text annotation. |
| Outcome: | The proposed tool is based on a fully modular architecture and provides three kinds of interfaces to annotate instances, evaluate annotation quality and manage the annotation task for annotators, reviewers and managers. |
Computer Assisted Annotation of Tension Development in TED Talks through Crowdsourcing (D19-59)
Copied to clipboard
| Challenge: | Using a neural network, we annotate whether tension is increasing, decreasing, or staying unchanged. |
| Approach: | They propose a machine-assisted method for the identification of tension development using a neural network based prediction model. |
| Outcome: | The proposed method is compared with other methods in in-house and crowdsourced environments. |
Mapping (Dis-)Information Flow about the MH17 Plane Crash (D19-50)
Copied to clipboard
| Challenge: | Digital media enables fast sharing of information, but also disinformation . studies on the spread of disinformation on social media focused on small, manually annotated datasets or used proxys for data annotation. |
| Approach: | They propose to use text classifiers to label Twitter content related to the MH17 crash to improve annotation accuracy. |
| Outcome: | The proposed classifier improves over a hashtag-based baseline, but still remains a challenge in labelling pro-Russian and pro-Ukrainian content with high precision. |
HUMAN: Hierarchical Universal Modular ANnotator (2020.emnlp-demos)
Copied to clipboard
| Challenge: | HUMAN is a web-based annotation tool that covers a variety of annotation tasks on textual and image data. |
| Approach: | They propose a web-based annotation tool that covers a variety of annotation tasks on textual and image data. |
| Outcome: | HUMAN covers a variety of annotation tasks on textual and image data and uses an internal deterministic state machine to chain different tasks in an interdependent manner. |
Model-based Annotation of Coreference (2020.lrec-1)
Copied to clipboard
| Challenge: | Annotators are asked to annotate coreferent spans of text, which is unnatural . we present an alternative in which annotators can preprocess documents and assign pronouns to entities. |
| Approach: | They propose an alternative in which annotators are asked to assign pronouns to entities and preprocess documents to create a knowledge base. |
| Outcome: | The proposed model-based approach leads to faster annotation and higher inter-annotator agreement and opens up an alternative approach to coreference resolution. |
Paladin: an annotation tool based on active and proactive learning (2021.eacl-demos)
Copied to clipboard
| Challenge: | Existing tools for active learning focus on the active learning algorithms and provide no user interface thus making it difficult to use for the end-users. |
| Approach: | They present an open-source web-based annotation tool for creating high-quality multi-label document-level datasets that integrates active learning and proactive learning. |
| Outcome: | The proposed tool is designed for multi-label annotation, but it can be adapted to other tasks in single-l Label settings. |
Semantic Span Annotation: An Exploratory Study of LLM Annotation (2026.acl-srw)
Copied to clipboard
| Challenge: | Structured span extraction research is siloed by context length, annotation task, and domain . Identifying a span within a natural language text and affixing it with a semantic label has been considered a core task in NLP . |
| Approach: | They propose a framework for structured span annotation that integrates five datasets under a common JSONL format with character-level offsets. |
| Outcome: | The proposed framework can generalize across four domains under three prompting configurations. |
Efficient Annotator Reliability Assessment with EffiARA (2025.acl-demo)
Copied to clipboard
| Challenge: | Obtaining annotations from experts is ideal, but this expertise is logistically and financially costly. |
| Approach: | They propose an annotation framework that supports the whole annotation pipeline from understanding the resources required for an annotation task to compiling the annotated dataset. |
| Outcome: | The proposed framework improves classification performance through annotator-reliability-based soft-label aggregation and sample weighting, and increases agreement among annotators through removal of identifying and replacing an unreliable annotation. |
Annotate the Way You Think: An Incremental Note Generation Framework for the Summarization of Medical Conversations (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing datasets for summarization of medical conversations are limited to conversation-summary pairs . a novel annotation framework is proposed to capture the summarizing process via an annotation task . |
| Approach: | They propose an incremental note generation framework that captures the human summarization process via an annotation task by instructing annotators to first incrementally create a draft note and polish it into a reference note. |
| Outcome: | The proposed framework shows that the human summarization process is much more efficient and accurate than the current method. |
A Multi-Axis Annotation Scheme for Event Temporal Relations (P18-1)
Copied to clipboard
| Challenge: | Existing temporal relation (TempRel) annotation schemes have low inter-annotator agreements even between experts, suggesting that the current annotation task needs a better definition. |
| Approach: | They propose to annotate temporal relation (TempRel) annotation schemes based on event start-points instead of a conventional 60’s-80’s model. |
| Outcome: | The proposed model improves IAA from the conventional 60’s to 80’s and can be used by crowdsourcing to alleviate labor intensity. |
GoodNewsEveryone: A Corpus of News Headlines Annotated with Emotions, Semantic Roles, and Reader Perception (2020.lrec-1)
Copied to clipboard
| Challenge: | Fewer studies address emotions as a phenomenon to be tackled with structured learning, which can be explained by the lack of relevant datasets. |
| Approach: | They propose to annotate 5000 English news headlines with their associated emotions, the corresponding emotion experiencers and textual cues, related emotion causes and targets, and the reader’s perception of the emotion of the headline. |
| Outcome: | The proposed method enables further research on emotion classification, emotion intensity prediction, emotion cause detection and supports qualitative studies. |
Confidence-based Ensembling of Perspective-aware Models (2023.emnlp-main)
Copied to clipboard
Silvia Casola, Soda Lo, Valerio Basile, Simona Frenda, Alessandra Cignarella, Viviana Patti, Cristina Bosco
| Challenge: | Human label variability has been a topic of research in the field of NLP recently . Exploiting disagreements in annotations has been shown to offer advantages for accurate modelling and fairer evaluation. |
| Approach: | They propose a highly perspectivist model that exploits disagreements in annotations to capture the subjectivity encoded in the annotation process. |
| Outcome: | The proposed model is validated on irony and hate speech detection scenarios in in-domain and cross-domain settings. |
Reward Modeling Requires Automatic Adjustment Based on Data Quality (2024.findings-emnlp)
Copied to clipboard
Binghai Wang, Rui Zheng, Lu Chen, Zhiheng Xi, Wei Shen, Yuhao Zhou, Dong Yan, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Reinforcement Learning from Human Feedback (RLHF) is a method for aligning language models with human values. |
| Approach: | They propose a method that automatically adjusts reward modeling based on data quality . they use preference data to train a reward model that is more aligned with human values . |
| Outcome: | The proposed method stabilizes reward model training and significantly improves alignment performance on human preference datasets. |
Can Humans Identify Domains? (2024.lrec-main)
Copied to clipboard
Maria Barrett, Max Müller-Eberstein, Elisa Bassignana, Amalie Brogaard Pauli, Mike Zhang, Rob van der Goot
| Challenge: | Textual domain is a crucial property within the Natural Language Processing community due to its effects on downstream model performance. |
| Approach: | They examine the level of human disagreement and the relative difficulty of each annotation task by training classifiers to perform the same task. |
| Outcome: | The authors show that human proficiency in identifying related intrinsic textual properties is low and that disagreements are high. |
Community-Driven Crowdsourcing: Data Collection with Local Developers (L18-1)
Copied to clipboard
| Challenge: | a community-driven approach to annotation applications and crowdsourcing programs is feasible, says a new study. |
| Approach: | They propose to partner with local developers to create custom annotation applications . they recruit and motivate crowd contributors from their communities to perform an annotation task . |
| Outcome: | The proposed approach combines local developers' knowledge of their social networks to collect labeled data. |
Charting the Linguistic Landscape of Developing Writers: An Annotation Scheme for Enhancing Native Language Proficiency (2024.lrec-main)
Copied to clipboard
| Challenge: | An annotation task was designed to capture orthographic, grammatical, lexical, semantic, and discursive patterns exhibited by college native English speakers participating in developmental education (DevEd) courses. |
| Approach: | They propose an annotation task to capture orthographic, grammatical, lexical, semantic, and discursive patterns exhibited by college native English speakers participating in developmental education courses. |
| Outcome: | The proposed annotation task captures orthographic, grammatical, lexical, semantic, and discursive patterns exhibited by college native English speakers participating in developmental education courses. |
CATs are Fuzzy PETs: A Corpus and Analysis of Potentially Euphemistic Terms (2022.lrec-1)
Copied to clipboard
| Challenge: | Euphemisms are a difficult topic because they are subject to language change and humans may not agree on what is a euphemist. |
| Approach: | They analyze a corpus of potentially euphemistic terms (PETs) and examples from the GloWbE corpus to examine their meanings. |
| Outcome: | The proposed corpus of potentially euphemistic terms and examples from the GloWbE corpus show that PETs generally decrease negative and offensive sentiment. |
Modelling Instance-Level Annotator Reliability for Natural Language Labelling Tasks (N19-1)
Copied to clipboard
| Challenge: | Existing models that estimate annotators' reliability only consider binary labels and multi-class labels. |
| Approach: | They propose an unsupervised model which can handle binary and multi-class labels and integrate neural networks to model the dependency between latent variables and instances. |
| Outcome: | The proposed model can handle binary and multi-class labels and can estimate reliability of annotators across instances. |
A Thesaurus-based Sentiment Lexicon for Danish: The Danish Sentiment Lexicon (2022.lrec-1)
Copied to clipboard
| Challenge: | a newly published Danish sentiment lexicon with a high lexical coverage was compiled using lexicographic methods and linked data. |
| Approach: | They propose to use lexicographic methods to compile a Danish sentiment lexicon with a high lexical coverage by linking words from a thesaurus to a comprehensive monolingual dictionary. |
| Outcome: | The proposed lexicon contains 13,859 Danish polarity lemmas and includes morphological information. |
Extending the gold standard for a lexical substitution task: is it worth it? (L18-1)
Copied to clipboard
| Challenge: | a lexical substitution task requires systems to identify words that are semantically close to the target and to select among candidates those that best fit the context. |
| Approach: | They propose to use a lexical substitution task to evaluate systems' performance . they use 300 sentences containing a target word and a second dataset based on the same data . |
| Outcome: | The proposed model is based on a set of 300 sentences containing a target word . the proposed model has not been evaluated to our knowledge . |
Validity, Agreement, Consensuality and Annotated Data Quality (2022.lrec-1)
Copied to clipboard
| Challenge: | a wide consensus is rife regarding the need for reference annotated datasets . however, the creation of such datasets is accompanied by theorectical and practical issues . |
| Approach: | They propose to use agreement among annotators as an indicator of consensus . they argue that it is difficult to produce gold-standard annotated datasets . |
| Outcome: | The proposed model focuses on the complex relations between agreement and reference and the emergence of consensus. |
A Two-Level Interpretation of Modality in Human-Robot Dialogue (2020.coling-main)
Copied to clipboard
| Challenge: | modal expressions are used to communicate and align world knowledge, but there is no obvious manner to ground them in the shared environment. |
| Approach: | They propose a two-level annotation scheme for modality that captures both content and intent and a task-oriented, pragmatic representation that maps to our robot's capabilities. |
| Outcome: | The proposed model can be grounded and dynamically interpreted. |
Ellogon Casual Annotation Infrastructure (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing annotation tools are desktop applications, allowing the annotation of corpora found on a single computer. |
| Approach: | They propose a new annotation paradigm, casual annotation, and propose an architecture and a reference implementation for the Ellogon Casual Annotation Tool. |
| Outcome: | The proposed paradigm and architecture have been evaluated for more than two years on an annotation task related to sentiment analysis. |
Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personas (2024.findings-emnlp)
Copied to clipboard
Salvatore Giorgi, Tingting Liu, Ankit Aich, Kelsey Isman, Garrick Sherman, Zachary Fried, João Sedoc, Lyle Ungar, Brenda Curtis
| Challenge: | Large language models (LLMs) are increasingly being used in human-centered social scientific tasks, such as data annotation, synthetic data creation, and engaging in dialog. |
| Approach: | They propose to prompt LLMs with human-like personas and ask them to answer as if they were a specific human, either explicitly, with exact demographics, political beliefs, and lived experiences, or implicitly via names prevalent in specific populations. |
| Outcome: | The proposed model is based on explicit, explicit, and implicit personas, and fails to show implicit biases. |
The Search for Agreement on Logical Fallacy Annotation of an Infodemic (2022.lrec-1)
Copied to clipboard
Claire Bonial, Austin Blodgett, Taylor Hudson, Stephanie M. Lukin, Jeffrey Micher, Douglas Summers-Stay, Peter Sutor, Clare Voss
| Challenge: | a parallel "infodemic" has emerged with the COVID-19 pandemic . logical fallacies can be subtly encoded in the structure of a document across multiple sentences . |
| Approach: | They evaluate an annotation schema for labeling logical fallacy types using linguist annotations . they propose to use a machine learning algorithm to train annotators for fallacy detection . |
| Outcome: | The proposed annotation schema is clear and non-overlapping for manual and system assignment. |
Conflicts, Villains, Resolutions: Towards models of Narrative Media Framing (2023.acl-long)
Copied to clipboard
| Challenge: | a growing body of work attempts to automatically detect media frames in the news or social media, but most adopts a topic-like view on frames, evading modelling the broader document-level narrative. |
| Approach: | They propose an annotation paradigm that breaks a complex annotation task into a series of simple binary questions. |
| Outcome: | The proposed method is both effective and transparent in its predictions. |
Annotation and Analysis of Extractive Summaries for the Kyutech Corpus (L18-1)
Copied to clipboard
| Challenge: | Summarization of multi-party conversation requires corpora to analyze characteristics of conversations and construct a method for summary generation. |
| Approach: | They propose to annotate a Japanese conversation corpus for a decision-making task . they compare extractive summarization methods with the annotated extractive summary . |
| Outcome: | The proposed corpus is the first annotated for conversation summarization tasks and freely available to anyone. |
An Attribution Relations Corpus for Political News (L18-1)
Copied to clipboard
| Challenge: | Existing resources for recognizing attributions in context are limited in size and completeness. |
| Approach: | They propose to use the largest and most complete attribution relations corpus to date . they propose to create sophisticated end-to-end solutions for attribution extraction . |
| Outcome: | The political news attribution relations corpus 2016 is the largest and most complete attribution relations corpuse to date. |
Zero-Shot Entailment Learning for Ontology-Based Biomedical Annotation Without Explicit Mentions (2025.coling-main)
Copied to clipboard
Rumana Ferdous Munne, Noriki Nishida, Shanshan Liu, Narumi Tokunaga, Yuki Yamagata, Kouji Kozaki, Yuji Matsumoto
| Challenge: | Automated biomedical annotation presents significant challenges when entities are not explicitly mentioned in the text. |
| Approach: | They propose an entailment-based zero-shot text classification approach to annotate biomedical text passages using the Homeostasis Imbalance Process (HOIP) ontology. |
| Outcome: | The proposed method performs well when processes are not explicitly mentioned . it is time-consuming and expensive to annotate biomedical texts with a specific ontology . |
Cross-Document, Cross-Language Event Coreference Annotation Using Event Hoppers (L18-1)
Copied to clipboard
| Challenge: | Defined event hoppers for the DEFT Rich Entities, Relations and Events (Rich ERE) annotation task. |
| Approach: | They propose an approach for cross-document, cross-lingual event coreference for the DEFT Rich Entities, Relations and Events (Rich ERE) annotation task. |
| Outcome: | The proposed approach is based on the definition of event hoppers for the DEFT rich entities, relations, events and their attributes . it yields 389 cross-document event hoppings in 505 documents in three languages . |
Impressions: Visual Semiotics and Aesthetic Impact Understanding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing image captioning and conditional generation models struggle to simulate plausible human responses to images. |
| Approach: | They propose a dataset to investigate the semiotics of images and how visual features and design choices can elicit specific emotions, thoughts and beliefs. |
| Outcome: | The proposed dataset improves existing models for image captioning and conditional generation. |
GPT is Not an Annotator: The Necessity of Human Annotation in Fairness Benchmark Construction (2024.acl-long)
Copied to clipboard
| Challenge: | Current benchmarks for social biases have limitations in scope, grounding, quality and human effort required. |
| Approach: | They propose to use a language model to help with the development of bias benchmarks . they extend previous work to a new community and set of biases: the Jewish community and antisemitism . |
| Outcome: | The proposed LLM does not perform well on the Jewish community and antisemitism task. |
Rethinking Annotation: Can Language Learners Contribute? (2023.acl-long)
Copied to clipboard
| Challenge: | Researchers have traditionally recruited native speakers to provide annotations for benchmark datasets, but there are languages for which recruiting native speakers is difficult. |
| Approach: | They recruit 36 language learners and provide two types of additional resources and perform mini-tests to measure their language proficiency. |
| Outcome: | The proposed method improves learners' language proficiency in terms of vocabulary and grammar. |
A Web-based Collaborative Annotation and Consolidation Tool (2020.lrec-1)
Copied to clipboard
| Challenge: | Annotation tools have a rigid structure, closed back-end and front-end, and are built in a non-user-friendly way rendering them unusable for a large cohort. |
| Approach: | They propose a web-based collaborative annotation and consolidation tool (AWOCATo) that supports varied textual formats and allows users to easily adapt to the annotation task. |
| Outcome: | AWOCATo supports a range of tasks and domains, filling the gap left by the lack of tools that can be used by people with and without programming knowledge. |
Annotating the Annotators: Analysis, Insights and Modelling from an Annotation Campaign on Persuasion Techniques Detection (2025.findings-acl)
Copied to clipboard
Davide Bassi, Dimitar Iliyanov Dimitrov, Bernardo D’Auria, Firoj Alam, Maram Hasanain, Christian Moro, Luisa Orrù, Gian Piero Turchi, Preslav Nakov, Giovanni Da San Martino
| Challenge: | Existing annotation campaigns based on heuristic guidelines have not been thoroughly discussed. |
| Approach: | They propose a probabilistic model for optimizing intervention scheduling to reduce the cost of an expert oversight in annotation tasks. |
| Outcome: | The proposed model advocates for an expert oversight in annotation tasks and periodic quality audits to reduce costs. |